Defenders of Wildlife is committed to becoming a more diverse, equitable, and inclusive organization and strives to be a leader in these efforts among environmental NGOs. The diversity, equity, inclusion and justice (DEIJ) working group was formed to help guide Defenders’ efforts along this path.
As a first step, the DEIJ working group conducted an organizational ‘listening tour.’ DEIJ Working Group members spoke with staff in each of Defenders’ departments and field offices to catalogue ongoing efforts related to DEIJ and identify where staff saw opportunities for growth in the organization. Defenders’ staff had many constructive thoughts, and distilling these ideas into actionable themes quickly became a challenge.
To help extract common themes from these responses, we decided to use a field of machine learning called ‘Natural Language Processing’ (NLP). The discipline of NLP covers a collection of techniques to automate the extraction of themes, sentiments, and associations from text. Our goal was to use this kind of analysis to help identify focal areas in which Defenders can advance our DEIJ practices in the coming year. This document will:
Statements were divided into efforts currently underway in the department, and opportunities that staff members saw for the organization to improve its DEIJ efforts. Because we’re most interested in setting goals for future work, in these analyses we focused only on the ‘Opportunity’ statements.
For these statements to be useful, we have to pre-process all the raw responses. We standardized language by making all characters lowercase, removing punctuation, and fixing contractions.
Because the goal of NLP is to identify themes and sentiments using the frequency of terms and their associations, it is helpful to remove common words that (usually) do not add meaning. There is a standard library of these words, and we added a few that also fit these criteria (e.g. ‘Defenders’, ‘Wildlife’, etc.). Finally, we ‘tokenize’ the data - breaking each statement into individual words or phrases. This resulted in 1868 single words.
Now that we have our listening tour statements broken down into individual (hopefully) meaningful tokens, the first thing we can do is look at the most common terms in these statements. Here are the top 10 single words.
## Selecting by nThese terms give us some idea of the common themes expressed in the listening tour, such as engagement with communities and training.
The frequencies of terms in listening tour responses has given us a good idea of where opportunities for DEIJ improvements across Defenders might exist. A more sophisticated analysis uses the same data to automatically identify clusters of terms. There are several techniques available for this task, and each uses the frequency of co-occurrence of different terms across responses to cluster terms together. We can then interpret the theme represented by each cluster by examining the words they contain.
Note: these methods usually require large (>10,000 records) datasets, and so our use here with 1868 data points is somewhat experimental.
But how many themes are present in the responses? The first step in this analysis is to group our data into progressively greater number of clusters, and use a measure of cohesion to see which number is most supported. Here we use the silhouette test, which compares the dissimilarity within clusters to that between clusters. Lower scores indicate stronger support, and we choose the cluster number (k) at which the rate of decline plateaus. This provides a balance between likelihood and simplicity of interpretation.
This graph tells us that the most likely number of thematic clusters for single terms is 7. Assuming 7 groups, we can use an algorithm called ‘Latent Dirichlet Allocation’ to assign terms to each of 7 clusers based on how frequenly different terms appear together. Plotting the terms most strongly associated with each cluster gives us an idea of the ‘themes’ captured in each cluster.
Clustering analyses require human interpretation to make sense of the clusters that emerge based on the words they contain. In this small dataset, some of the clusters are more ambiguous than others - however there appear to be several clear themes. For instance, Topic 1 contains terms related to community outreach and engagmenent. Topic 7 seems to include words related to personnel practices, and Topic 6 encapsulates expanding Defenders’ audience. Seeing the terms that co-occur frequently in staff responses is a starting point for understanding the common themes that were expressed.
Thus far, we’ve only looked at single terms to understand the topics present in the listening tour responses. It can also be informative to examine two word phrases, or bigrams. In the listening tour dataset, there were 497 instances of meaningful bigrams.
Looking at the most used two-word bigrams might provide some additional clarity around important topics.
The themes emerging based on the most common two-word phrases used during the listening tour are clearer. The working group will likely only take up 2 or 3 areas around which we will develop projects and initiatives, so how might we prioritize this list?
In addition to the raw frequency, it could be informative to see how these terms were distributed among departments. We can do this by plotting the number of departments that used a term against the overall frequency of that term. Ideas that Defenders may want to prioritize will be mentioned frequently and by many departments.
In fact, taking the bigrams that appear in the top right portion of this plot would provides common themes on which Defenders could focus expanding and improving DEIJ practices. Lets examine the responses from which these phrases originated to get some context. For example:
In theory, we can perform a clusering analysis for bigrams as well. Unfortunately, these clustering algorithms failed to converge using bigrams. This is likely because these kinds of analyses usually require large data sets, and the 497 bigrams recorded were insufficient.
However, Just by examining the statements associated with the most important bigrams, we identified:
Natural Language Processing is a useful tool for making sense of written text. These techniques still require some human interpretation, but we were able to use these techniques to quickly filter through a large amount of text and extract the most salient terms, sentiments, and themes. Using automated processes also provides a replicable method for identifying themes and priorities in the future, should Defenders conduct similar or more extended types of social response data.